Seeding replication

ABSTRACT

Seeding replication is disclosed. One or more but not all files stored on a deduplicated storage system are selected to be replicated. One or more segments referred to by the selected one or more but not all files are determined. A data structure is created that is used to indicate that at least the one or more segments are to be replicated. In the event that an indication based at least in part on the data structure indicates that a candidate segment stored on the deduplicating storage system is to be replicated, the candidate segment is replicated.

This application is a continuation of U.S. patent application Ser. No.11/807,204 entitled SEEDING REPLICATION filed May 24, 2007, now U.S.Pat. No. 7,827,137, which is incorporated herein by reference for allpurposes, which claims priority to U.S. Provisional Patent ApplicationNo. 60/925,624 entitled SEEDING REPLICATION filed Apr. 19, 2007, whichis incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

A replica system stores the same data or a portion of the same data asan originating system. The replica system can be used to recover datawhen data in the originating system is corrupted or lost. For efficiencyof storage, both the replica and the originating system may bededuplicating systems in which in-coming data is broken up intosegments, and if the segment is already stored on the system, areference to the already-stored segment is stored instead of storing thesegment again. Deduplication typically results in a substantial (e.g.,10×) reduction in the amount of space required to store data for thesystem.

When first starting replication from one system to another, if thereplica is to store all of the same data as the originating system, thenthe task is clear: transfer all the data over. This is efficient for adeduplicating system, since only the deduplicated segments and thereferences that enable file reconstruction need to be sent. However, ifthe replica is to store only a portion of the data on the originatorsystem, then it is not obvious which of the segments stored need to besent over to the replica. One simple solution is to run through the listof references to segments for the portion of the data to be stored onthe replica and ask the replica system if the referred to segment hasalready been stored. The segment is then only transmitted in the eventthat it is not already on the replica system. However, this requiresback and forth traffic for each reference in the list and checking bythe replica system for each reference of a segment. With deduplication,there may be many times more of such references than there are actualdata segments. It would be beneficial to be able seed replication for aportion of data stored on a deduplicated system without generatingtraffic and checking for each reference of a segment.

An analogous situation exists when copying a portion of the data storedon one deduplicated system to a second deduplicated system on a one-timebasis. All of the segments referenced by the portion of the data beingcopied need to be sent to the second system. However, as above, checkingeach reference to see if the corresponding segment is to be sent to asecond system, can create substantial traffic between the two systemsfor each reference that needs to be checked. It would be beneficial tobe able to copy a portion of data stored on a deduplicated systemwithout generating traffic and checking for each reference of a segment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a system forseeding replication.

FIG. 2A is a block diagram illustrating an embodiment of a portion of adata stream or data block.

FIG. 2B is a block diagram illustrating an embodiment of segmenting aportion of a data stream or a data block.

FIG. 3 is a block diagram illustrating an embodiment of file references.

FIG. 4A is a block diagram illustrating an embodiment of a datastructure.

FIG. 4B is a block diagram illustrating an embodiment of a datastructure.

FIG. 4C is a block diagram illustrating an embodiment of a datastructure.

FIG. 5 is a flow diagram illustrating an embodiment of a process forseeding replication.

FIG. 6 is a flow diagram illustrating an embodiment of a process fordetermining referenced segments.

FIG. 7A is a flow diagram illustrating an embodiment of a process foradding to a data structure.

FIG. 7B is a flow diagram illustrating an embodiment of a process foradding to a data structure.

FIG. 7C is a flow diagram illustrating an embodiment of a process foradding to a data structure.

FIG. 8A is a flow diagram illustrating an embodiment of a process forreplicating a segment.

FIG. 8B is a flow diagram illustrating an embodiment of a process forreplicating a segment.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess, an apparatus, a system, a composition of matter, a computerreadable medium such as a computer readable storage medium or a computernetwork wherein program instructions are sent over optical orcommunication links. In this specification, these implementations, orany other form that the invention may take, may be referred to astechniques. A component such as a processor or a memory described asbeing configured to perform a task includes both a general componentthat is temporarily configured to perform the task at a given time or aspecific component that is manufactured to perform the task. In general,the order of the steps of disclosed processes may be altered within thescope of the invention.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

Seeding replication is disclosed. In a deduplicated system, seedingreplication for a portion of the data sharing a common pool of segmentson an originating system requires determining the segments referencedthat enable the reconstruction of the portion of data. A list or a datastructure can be generated that contains information regardingreferenced segments. For unique segments stored on the originatorsystem, each can be checked using the list or data structure to see ifthe segment has been referenced by the portion of data. If the segmenthas been referenced, the segment is sent to the replica system. Thisallows the communications back and forth between the originating systemand the replica system to be reduced to the segment level instead of thereference level. Also, in the case of the data structure, efficient useof memory is possible, since the vector is substantially smaller thanthe list of unique referenced segments. In addition, the list or thedata structure is generated on the originator system without anycommunication with the replica system saving communication bandwidthbetween the originator and replica systems.

In some embodiments, multiple files will refer to the same storedsegment because the content of a segment in the different files is thesame. This is in contrast to a situation where multiple versions of thesame file are stored on the same system (e.g., when multiple snapshotsof a system are stored on one system) where each of the multipleversions of the same file refer to the same segment. A file mayimplement a file in a file system, a logical block device, a virtualtape cartridge, a database table, or any other kind of data object or aportion of such a data object that a data storage system may store.

FIG. 1 is a block diagram illustrating an embodiment of a system forseeding replication. In the example shown, storage system 100 storesdata for clients represented in FIG. 1 by client 112 and client 108. Aclient may have a local storage device in addition to local memory. Forexample, client 108 has storage 110 whereas client 112 does not have alocal storage device. Storage system 100 stores data either usinginternal storage device 102 or attached external storage device 104.

Storage system 100 communicates with clients via network 106. Network106 comprises one or more of the following: a local area network, a widearea network, a wired network, a wireless network, the Internet, a fibernetwork, or any other appropriate network enabling communication.Storage system 100 communicates with replica system 120 via network 106as well. Replica system 120 includes internal storage device 122 and isconnected to external storage device 124.

Replica system 120 stores all or a portion of data stored on storagesystem 100. Initially, all or the portion of data stored on storagesystem 100 is replicated on replica system 120 by sending appropriatedata via network 106. After the initial seeding, replica system 120 isupdated by sending from storage system 100 new data. Updating can becontinuous, sent in bursts on a regular schedule, when the amount ofdata exceeds a certain size, or at other appropriate times. In the caseof data copying, the initial seeding completes the task and there is nosubsequent updating.

In various embodiments, storage devices 102, 104, 122, and 124 comprisea single storage device such as a hard disk, a tape drive, asemiconductor memory, a plurality of storage devices such as a redundantarray system (e.g., a redundant array of independent disks (RAID)), asystem for storage such as a library system or network attached storagesystem, or any other appropriate storage device or system.

FIG. 2A is a block diagram illustrating an embodiment of a portion of adata stream or data block. In the example shown, portion of data streamor data block 200 is shown. Portion of data stream or data block 200 isreceived by and stored on a storage system such as storage system 100 ofFIG. 1. Portion of data stream or data block 200 is sent from one ormore clients such as client 108 or 112 of FIG. 1.

FIG. 2B is a block diagram illustrating an embodiment of segmenting aportion of a data stream or a data block. In the example shown, theportion of the data stream or data block is segmented into a pluralityof segments represented in FIG. 2B by 202, 204, and 206. The data streamor data block is segmented by creating a plurality of segments from thedata stream or data block that can be used to reconstruct the datastream or data block. Segments, when used to reconstruct the data streamor data block, can be overlapping, non-overlapping, or a combination ofoverlapping and non-overlapping. Segment boundaries can be determinedusing file boundaries, directory boundaries, byte counts, content-basedboundaries (e.g., when a hash of data in a window is equal to a value),or any other appropriate method of determining a boundary.Reconstruction of a data block, data stream, file, or directory includesusing references to the one or more segments that originally made up thedata block, data stream, file, or directory. Segments such as 202, 204,and 206 are stored in storage device 210—for example, as stored segments212, 214, and 216 respectively.

FIG. 3 is a block diagram illustrating an embodiment of file references.In some embodiments, file references can be used to reconstruct a filefrom segments stored in a storage device such as storage device 210. Inthe example shown, file 300 can be reconstructed using segmentreferences 302, 304, 306, and 308. File 310 can be reconstructed usingsegment references 312, 314, 316, and 318. Offsets are stored to enablethe reconstruction of files using segment references in the case thatthere is an overlap or only a portion of the segment reference is usedin the reconstruction of the file. A similar referencing structure canbe used for a data block, a data stream, a directory, or any other unitof data to enable their reconstruction using segments stored. In variousembodiments, a segment reference comprises a content-based reference, anon-content-based reference, a fingerprint, or any other appropriatereference for a segment.

FIG. 4A is a block diagram illustrating an embodiment of a datastructure. In some embodiments, the data structure of FIG. 4 is used toindicate if a segment was referenced to determine if the segment shouldbe replicated. In the example shown, the data structure 400 includes aplurality of bit locations represented by 401, 402, 404, 406, 408, and410. The bit locations are initialized to ‘0’. For a segment that isreferenced, a plurality of hashes of the segment reference arecalculated—for example, in FIG. 4A hashes of the segment reference arerepresented by Hash(Seg Ref) 412, 414, 416, and 418. The value of thehash is used to point to a bit location in data structure 400. The bitlocation pointed to is turned to a ‘1’. To check if a segment isindicated for replication, the same hashes of a reference to the segmentare computed and if all the bits in the locations pointed to by thehashes have been set to a ‘1’, then the segment is presumed to bereferenced and should therefore be replicated. In some cases, a segmentmay appear to be referenced because all bit locations have been set to a‘1’ even though a reference to the segment was not indicated. Suchcollisions occur when a combination of other segments indicated happento set all the locations to a ‘1’. The length of data structure and thenumber of hash functions is selected to balance the space required forthe data structure, the computation required for the hash functions, andthe number of tolerable collisions in the data structure for thesegments. It is desirable that data structure 400 be able to indicatereasonably accurately whether a given segment has been referenced sothat few segments that do not need to be replicated are replicated. Invarious embodiments, data structure 400 comprises a Bloom filter or asimilar data structure based on hashes.

FIG. 4B is a block diagram illustrating an embodiment of a datastructure. In the example shown, data structure 420 can be used todetermine a list of inexact references to segments. A plurality ofshortened references is represented in FIG. 4B by 422 and 424. Shortenedreference 422 is created from segment reference 430 and placed in datastructure 420. Shortened reference 424 is created from segment reference432 and placed in data structure 420. For example, a shortenedfingerprint such as n-bytes (e.g., 4) of a m-byte (e.g., 24) segmentreference can be used as an inexact reference where m is less than n.Segments with references that match in the n-bytes of any of theshortened fingerprints of the data structure are added to a list ofsegments that are to be sent over from an originator system to a replicasystem. The use of the n-byte match ensures that necessary segments arereplicated. There may be some additional segments transferred. On theother hand, memory space is reduced in using a shortened (e.g., a4-byte) comparison as opposed to a full (e.g., 24-byte) comparison.

FIG. 4C is a block diagram illustrating an embodiment of a datastructure. In the example shown, data structure 440 can be used todetermine a list of exact references to segments. A plurality ofreferences is represented in FIG. 4C by 442 and 444. Reference 442 iscreated from segment reference 450 and placed in data structure 440.Reference 444 is created from segment reference 452 and placed in datastructure 440. The data structure of exact segments referenced can beused in the process of seeding replication in a replica.

FIG. 5 is a flow diagram illustrating an embodiment of a process forseeding replication. In some embodiments, the process of FIG. 5 is usedto seed a replica system such as replica system 120 of FIG. 1. In someembodiments, the process of FIG. 5 is used to copy a set of data fromone system to another. In the example shown, in 500 file(s) is/areselected to be replicated. In various embodiments, file(s) is/areselected using selection of one or more files in a graphic userinterface, using a list of files, by clicking in a window, by selectinga directory, by selecting a plurality of directories, using a list ofdirectories, or any other appropriate method of selecting files. Invarious embodiments, the file(s) may be determined automatically by thesystem for the purpose of load balancing, meeting quality of servicetargets or other reasons. In various embodiments, the files to betransferred may implement logical block devices, virtual tapecartridges, or other kinds of sets of data or portions of such sets ofdata. In 502, file metadata and segment references for selected filesare replicated and segment references are added to a data structure usedto indicate that at least determined segments are to be replicated. Insome embodiments, more segments than those necessary for reconstructionof the selected file(s) are indicated to be replicated. In 506, a firstcandidate segment stored on a deduplicating system is selected. In 508,it is determined if the data structure indicates that the selectedcandidate segment is to be replicated. If the selected candidate segmentis to be replicated, then in 510 the selected candidate segment isreplicated and control passes to 512. If not, control passes to 512. In512, it is determined if there are more candidate segments. If so, thenin 514 the next candidate segment is selected and control passes to 508.In some embodiments, when possible the next candidate segment selectedin 514 is one stored (e.g., on the deduplicating storage system) inclose proximity to the previous segment for efficient access. If thereare no more segments in 512, then the process ends. In some embodiments,every segment stored in the deduplicating storage system is a candidatefor replication. In some embodiments, the candidate segments areprocessed substantially in the order they are located in thededuplicating storage system. This can provide efficiencies by reducingthe time spent accessing the segments (e.g., accessing segments inbatches or not requiring substantial seeks before a next segmentaccess).

FIG. 6 is a flow diagram illustrating an embodiment of a process fordetermining referenced segments. In some embodiments, the process ofFIG. 6 is used to implement 502 of FIG. 5. In the example shown, in 600a first file is selected. In 601, the selected file's metadata (e.g.,file name, file size, file create date, etc.) is replicated. In 602, afirst reference of the selected file is selected. In 604, the referenceis replicated. In 606, segment reference is added to a data structureused to indicate that at least determined segments are to be replicated.In 608, it is determined if there are more references. If so, then in610 a next reference is selected and control passes to 604. If not, thenin 612 it is determined if there are any more files. If so, then in 614a next file is selected and control passes to 602. If not, the processends.

FIG. 7A is a flow diagram illustrating an embodiment of a process foradding to a data structure. In some embodiments, the process of FIG. 7Ais used to implement 606 of FIG. 6. In the example shown, in 700 it isdetermined if the segment referred to by the selected reference isalready in an exact list. If so, then the process ends. If not, then in702 the segment is added to the exact list. The exact list comprises alist of unambiguous references to segments. For example, a list ofsegment fingerprints or other kinds of references. In variousembodiments, the list is generated in such a way as to eliminate or noteliminate duplicate segments appearing on the list, and/or at a timewhen storing the original files on the originator storage system or at alater time as needed. In some embodiments, the list is sorted by segmentreference. In some embodiments, the list is sorted by storage location.

FIG. 7B is a flow diagram illustrating an embodiment of a process foradding to a data structure. In some embodiments, the process of FIG. 7Bis used to implement 606 of FIG. 6. In the example shown, in 710 aplurality of hashes is computed of the selected reference. In 712, eachof the plurality of hashes is used to change the data structure, ifappropriate. If the location has already been changed (e.g., set to‘1’), then the location is not changed. In some embodiments, the hashvalue is used to determine a bit location in the data structure to setto a ‘1’ value. The data structure can be used to determine if a segmentwas referenced.

FIG. 7C is a flow diagram illustrating an embodiment of a process foradding to a data structure. In some embodiments, the process of FIG. 7Cis used to implement 606 of FIG. 6. In the example shown, in 720 ashortened reference is created. In 722, the shortened reference is addedto the data structure. In some embodiments, the data structure is a longlist of shortened references. The data structure can be used todetermine if a segment was likely referenced.

In some embodiments, the data structure comprises the list of segmentsto be replicated as generated by the process of FIG. 6.

FIG. 8A is a flow diagram illustrating an embodiment of a process forreplicating a segment. In some embodiments, the process of FIG. 8A isused to implement 510 of FIG. 5. In the example shown, in 800 a segmentis sent to the replica system. In this example, the operatingassumptions are that the replica system does not already have a copy ofthe segment and that the segment successfully reaches the replicasystem. In some embodiments, an acknowledgement is received indicatingthat the segment has been received and/or stored on the replica system(not shown).

FIG. 8B is a flow diagram illustrating an embodiment of a process forreplicating a segment. In some embodiments, the process of FIG. 8B isused to implement 510 of FIG. 5. In the example shown, in 810 a segmentidentifier is sent to the replica system. In 812, an indication isreceived if the segment is stored on the replica system. In 814, it isdetermined if the indication indicates that the segment is stored on thereplica system. If so, then the process ends. If not, then in 816 thesegment is sent to the replica system. In some embodiments, anacknowledgement is received indicating that the segment has beenreceived and/or stored on the replica system (not shown).

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A computerized system for seeding replicationcomprising: one or more computer processors; a computer memory coupledto the one or more computer processors to provide the one or morecomputer processors with instructions configured to: select one or morefiles stored on a deduplicated storage system to be replicated, whereinthe selected one or more files do not include all files stored on thededuplicated storage system; determine one or more segments referred toby the selected one or more files, wherein one segment of the one ormore segments is used to reconstruct more than one file of the selectedone or more files; create a data structure that is used to indicate thatat least the one or more segments are to be replicated, wherein the datastructure is created based at least in part on the determined one ormore segments referred to by the selected one or more files; determinethat a candidate segment of a plurality of candidate segments stored onthe deduplicated storage system is to be replicated based at least inpart on the data structure, wherein the plurality of candidate segmentsare processed based at least in part on a storage location of theplurality of candidate segments in the deduplicated storage system. 2.The system as in claim 1, wherein the one or more segments are referredto using a content-based reference of the selected one or more files. 3.The system as in claim 1, wherein the one or more segments are referredto using a non-content-based reference of the selected one or morefiles.
 4. The system as in claim 1, wherein the one or more segments arereferred to using a fingerprint of the selected one or more files. 5.The system as in claim 1, wherein the data structure comprises an exactlist of the one or more segments.
 6. The system as in claim 5, whereinthe exact list of the one or more segments is deduplicated.
 7. Thesystem as in claim 5, wherein the exact list of the one or more segmentsis sorted.
 8. The system as in claim 1, wherein the data structurecomprises a probabilistic data structure.
 9. The system as in claim 8,wherein the probabilistic data structure comprises an inexact list ofthe one or more segments.
 10. The system as in claim 1, wherein thecreation of the data structure includes using hash functions.
 11. Thesystem as in claim 1, wherein the data structure is constructed at asame time as segments are stored on the deduplicated storage system. 12.The system as in claim 1, wherein the data structure is constructed inresponse to a request to seed a replica system.
 13. The system as inclaim 1, wherein replicating the candidate segment includes one or moreof the following: sending the candidate segment to a replica system,receiving an acknowledgement that the candidate segment was received orstored on the replica system, sending a reference to the replica systemto determine whether the candidate segment referred to by the referenceis already stored on the replica system, and sending the candidatesegment in response to determining that the candidate segment is notalready stored on the replica system.
 14. The system as in claim 1,wherein the one or more computer processors are further configured toreplicate one or more references by the selected one or more files. 15.The system as in claim 14, wherein the selected one or more files arereconstructed based at least in part on the one or more references. 16.The system as in claim 1, wherein every segment stored in thededuplicated storage system is a candidate for replication.
 17. Thesystem as in claim 1, wherein the one or more computer processors arefurther configured to process the plurality of candidate segments in anorder such that at least two successive candidate segments processed arestored in close proximity in the deduplicated storage system.
 18. Acomputer program product for seeding replication, the computer programproduct being embodied in a non-transitory computer readable storagemedium and comprising computer instructions for: selecting one or morefiles stored on a deduplicated storage system to be replicated, whereinthe selected one or more files do not include all files stored on thededuplicated storage system; determining one or more segments referredto by the selected one or more files, wherein one segment of the one ormore segments is used to reconstruct more than one file of the selectedone or more files; creating a data structure that is used to indicatethat at least the one or more segments are to be replicated, wherein thedata structure is created based at least in part on the determined oneor more segments referred to by the selected one or more files;determining that a candidate segment of a plurality of candidatesegments stored on the deduplicated deduplicating storage system is tobe replicated based at least in part on the data structure, wherein theplurality of candidate segments are processed based at least in part ona storage location of the plurality of candidate segments in thededuplicated storage system.
 19. A method utilizing one or more computerprocessors for seeding replication, wherein the method comprising:selecting one or more files stored on a deduplicated storage system tobe replicated, wherein the selected one or more files do not include allof the files stored on the deduplicated storage system; determining,using one or more computer processors, one or more segments referred toby the selected one or more files, wherein one segment of the one ormore segments is used to reconstruct more than one file of the selectedone or more files; creating a data structure that is used to indicatethat at least the one or more segments are to be replicated, wherein thedata structure is created based at least in part on the determined oneor more segments referred to by the selected one or more files;determining that a candidate segment of a plurality of candidatesegments stored on the deduplicated storage system is to be replicatedbased at least in part on the data structure, wherein the plurality ofcandidate segments are processed based at least in part on a storagelocation of the plurality of candidate segments in the deduplicatedstorage system.